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ABSTRACT 

Critical thinking is considered a necessary learning outcome for all college students and essential 
for academic and career success. There are many challenges to developing a comprehensive 
approach to teaching and assessing critical thinking skills. Although the literature has many 
examples of the incorporation of critical thinking and assessment into courses, longitudinal 
studies following engineering students through their undergraduate career are lacking. This study 
assessed the impact of using the Paid-Elder Critical Thinking Framework to enhance 
undergraduate students ’ critical thinking skills with the hypothesis: There will be a significant 
increase in undergraduate students ’ critical thinking abilities from the freshman to the senior year 
with the explicit and strategic incorporation of critical thinking assignments. The research 
question was, “How do the critical thinking skills of undergraduate engineering students change 
as they progress through the engineering program with the explicit and strategic incorporation of 
critical thinking assignments? ” The study was a descriptive, longitudinal study of three 
engineering student cohorts as they progressed through the four year undergraduate program. 

The study was approved by the university’s Institutional Review Board. There was a statistically 
significant increase in critical thinking scores over the four years for each of the three cohorts. 
Integrating and evaluating critical thinking assignments into engineering curricula is possible, but 
a major challenge to critical thinking assessment using a holistic rubric is training engineering 
faculty in their use. The results are encouraging, and participating faculty agree; but sustaining 
these efforts to imbed critical thinking assignments throughout the engineering college curriculum 
will require effort and administrative support. 

Keywords: Critical Thinking Assessment; Paul-Elder Critical Thinking Framework; Longitudinal Cohort Group 

Study; Undergraduate Engineering 


INTRODUCTION 

longitudinal study to enhance the critical thinking skills of their undergraduate students was 
undertaken at the school of engineering at a large, urban research university. The study was based 
on three student cohorts as they progressed through the four year undergraduate program. Initial 
results from the first cohort have been reported previously (Ralston & Bays, 2013). Included in this article are: 
background for the study, supporting literature on critical thinking and on the Paul-Elder framework (Paul & Elder, 
2009) as the vehicle to teach critical thinking, summary of the development of the overall study including the 
creation of a holistic rubric aligned with the Paul-Elder critical thinking framework for use with undergraduate 
engineering students, discussion of the faculty training required for both teaching the framework and rating the 
written artifacts, and discussion of the results from the assessment of students’ critical thinking skills over the entire 
study period. 
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BACKGROUND 

In 2007, the university began implementing a multi-year Quality Enhancement Plan to enhance 
undergraduate students' critical thinking skills in all undergraduate programs across the entire undergraduate 
experience as part of the regional reaccreditation process. For the reaccreditation plan, the university selected 
Scriven and Paul’s (1987) definition that critical thinking “is the intellectually disciplined process of actively and 
skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or 
generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action .” A 
primary goal early in the initiative was to identify a critical thinking framework that could be used across the 
undergraduate programs for consistency in terminology and assessment. Frameworks and models reviewed for the 
initiative included: (Bloom, 1956), (Bogo & Vayda, 1998), (Brookfield, 1987), (Facione & Facione, 1998), and 
(Paul & Elder, 2001, 2009). The models were reviewed for their comprehensiveness, applicability to all 
undergraduate programs, inclusion of specific cognitive skills (e.g., metacognition), and availability of high quality 
resources. Faculty, administrators and staff responsible for leading the initiative unanimously agreed to select the 
Paul-Elder critical thinking framework (Paul & Elder, 2009) to provide intentional and consistent language across 
undergraduate programs because of the framework’s comprehensiveness, discipline neutral terminology and 
extensive high quality resources. 

The Engineering School embraced the comprehensive inclusion of critical thinking assessment because 
engineering faculty felt, and research supports that critical thinking is foundational to engineering education and to 
engineering practice (Mattingly, Weatherton, Druzic, Frost, & Rahman, 2014; Alfrey & Cooney, 2009; Rogers, 
2006). However, faculty responses also indicated that critical thinking was not being included as an explicit 
component in preparation and implementation in most course lectures and syllabi. Faculty members typically 
assumed that students could learn to think critically by watching them demonstrate critical thinking as they worked 
problems or discussed design issues. After more in-depth discussions, most faculty members agreed that they never 
explicitly discussed the thought processes behind problem investigation, analysis of situations, synthesis of designs, 
and evaluation of alternatives. As a result, the Engineering School embraced the Paul-Elder Framework as a means 
of introducing common labels and descriptions for all discussions of critical thinking and developed a program to 
inculcate critical thinking, overtly and intentionally throughout the undergraduate curriculum, culminating with the 
engineering capstone design experiences. Of the approximately 90 faculty members, 25-30 were actively engaged 
in this longitudinal study each year. A key challenge is continuing this effort and maintaining a focus on integration 
of critical thinking throughout the curriculum now the study has concluded. 

Although not mentioned explicitly in the Accreditation Board for Engineering and Technology (ABET) 
outcomes, one thoroughly versed in the Paul-Elder framework could argue that critical thinking is actually the 
foundation for all of the eleven program outcomes assessed for accreditation. Engineering programs must 
demonstrate that their students attain: (a) an ability to apply knowledge of mathematics, science, and engineering, 
(b) an ability to design and conduct experiments, as well as to analyze and interpret data,(c) an ability to design a 
system, component, or process to meet desired needs within realistic constraints such as economic, environmental, 
social, political, ethical, health and safety, manufacturability, and sustainability, (d) an ability to function on 
multidisciplinary teams, (e) an ability to identify, formulate, and solve engineering problems, (1) an understanding of 
professional and ethical responsibility, (g) an ability to communicate effectively, (h) the broad education necessary 
to understand the impact of engineering solutions in a global, economic, environmental, and societal context, (i) a 
recognition of the need for, and an ability to engage in life-long learning, (j) a knowledge of contemporary issues, 
(k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice (ABET, 
2013). All outcomes require “the intellectually disciplined process of actively and skillfully conceptualizing, 
applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, 
experience, reflection, reasoning, or communication, as a guide to belief or action’’, i.e. critical thinking (Scriven & 
Paul, 1987). Many engineering educators believe life-long learning involves critical thinking, including (Shuman, 
Besterfield-Sacre, & McGourty, 2005) and (Jiusto & DiBiasio, 2006). The assessment of critical thinking 
complements the assessment of ABET outcomes for engineering programs. The better thinkers students are, the 
better equipped students are to achieve the outcomes. Students must be able to think critically as individuals and as 
members of teams. The assessment of program outcomes for engineering schools will be enhanced by strong, well- 
assessed critical thinking programs. 
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Assessment of critical thinking in general is difficult and there is not a consensus on how to measure it (Ku, 
2009), (Liu, Frankel, & Roolir, 2014). One study (Renaud & Murray, 2008) suggested that improvement in critical 
thinking skills are more clearly detected with items focusing on specific course content rather than on general 
content. Although program assessment of critical thinking has been undertaken in other fields (Bensley & Murtagh, 
2012; Cavaliere & Mayer, 2012; McKitrick & Barnes, 2012), the only published discussion of program assessment 
of critical thinking for engineering was the recent work by Eppes and colleagues (Eppes, Milanovic, & Sweitzer, 
2012). Their work presents an assessment framework for five liberal education outcomes, including critical thinking, 
in the University of Hartford’s engineering and technology programs, but gives results from only a pilot study 
involving a single capstone design course. This work offers an example of comprehensive inclusion and assessment 
of critical thinking throughout the undergraduate engineering program using specific course content. 

FRAMEWORK AND SUPPORTING LITERATURE 

Since 1993, 167 articles in the Journal for Engineering Education have mentioned critical thinking, and 
249 papers in the 2014 National ASEE Conference included the words, clearly indicating recognition of a growing 
interest in developing this ability in students. NSF has supported the development of instruments to assess students’ 
critical thinking skills and many faculty in all disciplines regard critical thinking as the most important outcome of 
an undergraduate education, (Stein & Haynes, 2011). In a 2011 survey conducted by the Association for American 
Colleges and Universities (AAC&U,2011), 95% of the chief academic officers from 433 institutions rated critical 
thinking as one of the most important intellectual skills for their students, and 81% of the employers surveyed 
wanted colleges to place a stronger emphasis on critical thinking. More recently, 93% of 318 employers surveyed 
agreed that, “a candidate’s demonstrated capacity to think critically, communicate clearly, and solve complex 
problems is more important than their undergraduate major, ” (Hart Research Survey, 2013). Although in at least 
one survey, employers perceived engineering graduates to be particularly poor at critical and independent thinking 
(Nielson, 2000). 

The term “critical thinking” is familiar to most engineering educators, but it is difficult to define easily. In 
one study, (Paul, Elder, & Bartell, T, 1996) it was found that 89% of teachers interviewed claimed critical thinking 
to be an important education objective, but only 19% were able to give a clear explanation of critical thinking. 
Ennis (1962) defined critical thinking as: “Critical thinking is reasonable, reflective thinking that is focused on 
deciding what to believe or do.” Scriven and Paul (1987) gave a more detailed definition: “Critical thinking is the 
intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or 
evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or 
communication, as a guide to belief or action.” Halpem in her 2003 book described critical thinking as “the thinking 
that is purposeful, reasoned and goal directed - the kind of thinking involved in solving problems, formulating 
inferences, calculating likelihoods, and making decisions, when the thinker is using skills that are thoughtful and 
effective for the particular context and type of thinking task”. Despite a wide variety of definitions, three key 
elements of critical thinking are reason, reflection, and judgment. Fundamentally, critical thinking is thinking about 
thinking, a meta-cognitive process. The combination of reflection and reason leads to the final element, belief in the 
validity of a premise, process or solution to a problem, which also can lead to action. Critical thinking develops 
conclusions by deducing or inferring answers to questions and then reflecting on the quality of the reasoning; the 
end result is action, based on those conclusions. 

Bailin and colleagues (Bailin, Case, Coombs & Daniels, 1999) point out that much literature at that time 
characterized critical thinking simply as one or more skills, mental processes, or sets of procedures and that this 
characterization led to the view that critical thinking could be taught simply by practicing or demonstrating it. They 
point out that critical thinking must be characterized in terms of specific performance criteria, that critical judgment 
is developed through applying knowledge in many contexts, and that improvement is made with frequent feedback 
and evaluation with respect to the quality of thinking demonstrated. The development of quality reasoning cannot 
be learned just by drill and practice in specific content areas, but requires opportunities for reflection and feedback 
as more involved thinking is required in a variety of contexts. Using critical thinking, or even demonstrating critical 
thinking, is not the same as teaching students to think critically. A common vocabulary and conceptual constructs 
on critical thinking can help address the challenges of teaching critical thinking. 


Copyright by author(s); CC-BY 


87 


The Clute Institute 



American Journal of Engineering Education - December 2015 


WolumeS. J htmbeeW. 


The Paul-Elder framework of critical thinking (Paul & Elder, 2008) chosen for this study has a formal 
structure and is a discipline-neutral schema (shown in Figure 1). It also addresses the concerns of Bailin and 
colleagues (Bailin, Case, Coombs & Daniels, 1999). The framework depicts critical thinking by applying Universal 
Intellectual Standards to the evaluation of typical Elements of Thought, with the goal of developing certain Essential 
Intellectual Traits in the thinker. The framework allows for the analysis and evaluation of thought, but more 
importantly, it provides a common vocabulary for those who want to discuss, evaluate, or teach critical thinking. 
The framework has been discussed specifically in light of engineering education (Paul et ah, 2006). The operational 
focus in the framework is the eight Elements of Thought which clarifies the building blocks of thinking; these 
building blocks are used by anyone who examines, analyzes, and reflects on intellectual work. These elements are 
embodied in eight categories of questions crucial to critical thinking: What is the purpose? (of the exercise, 
discussion or argument), What is the point of view? (of each participant, group or entity), What are the assumptions? 
(inherent premises of the argument), What are the implications? (of the reasoning or assumptions), What 
information is missing and needed to reach a conclusion? What inferences are being made? What is the most 
fundamental concept put forward by participants?, and What is the specific question that is being answered? The 
Universal Intellectual Standards are the criteria used to evaluate the quality of the thinking. According to the 
framework, applying the standards to the elements is what transforms common/general/everyday thinking to critical 
thinking. 


Figure 1 . The Paul-Elder Critical Thinking Framework 



Cooney et al. (2008) found a disturbing dissonance between the degree to which engineering and 
technology faculty at IUPUI believed they were incorporating critical thinking experiences in their courses, and the 
amount of critical thinking experience students perceived they were receiving. Analysis of faculty anecdotes and 
examples of teaching and learning indicated to Cooney and colleagues that a clear disconnect existed between what 
teachers considered critical thinking experience and what students identified as exercises in critical thinking. Cooney 
and colleagues reviewed research on developing critical thinking skills in engineering and technology students, and 
identified two significant activities that could be incorporated into classes easily and were very effective: writing 
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based on reflection and learning through solving problems. In any type of writing activity, students digest given 
information, analyze the content critically, analyze the reasoning incorporated in the information, think about their 
own thinking, and then articulate their thoughts and/or value judgments. One method to implement such writing for 
reflection is to provide rubrics for evaluating open-ended problems, and then to provide feedback to help students 
revise their initial written reflections. The feedback is an important step in refining thinking and cognition. Other 
research evaluating critical thinking using writing exercises in engineering education can be found in (Lunt & Helps, 
2001), (Gurmen, Lucas, Malmgren, & Folger, 2003) and (Damron & High, 2008). “Critical thinking skills allow 
students to more quickly assimilate subject-specific course content and also provide a framework that allows 
students to engage and respond to less-well defined problems”, (Tsui, 2000, 2002). Tsui’s research from four 
undergraduate institutions indicated that not only did writing assignments that required students to demonstrate a 
synthesis of material, evaluate arguments, and deduce conclusions improve critical thinking skills, but “re-writing” 
was even better due to the direct inclusion of feedback. 

Problem solving is the quintessence of engineering, and learning through solving problems has been 
recognized for many years as a primary way to teach critical thinking (Bailey, 1979). Tsui (1998) reviewed studies 
of how critical thinking has been taught and found that problem solving is especially effective. Engineering faculty 
have focused on developing critical thinking based on problem solving, especially since the adoption of the ABET 
2000 criteria (Mani, Omidvar, & Knott, 2003), (Lombardo, 2004), and (Papadopoulos, Rahman, & Bostwick, 2006). 
Recently, ethics issues and critical thinking were intertwined in a problem-based learning case-study (Chang & 
Wang, 2011). 

Other research related to the development of critical thinking in engineering involves ethical decision 
making (Wolverton & Wolverton, 2003), (Swalie & Kreppel, 2001), conducting experiments, (Bruno & Anderson, 
2005), assessing the social impact of technology (Nelson, 2001), using the backward design process to modify 
courses to emphasize critical thinking (Sgro & Freeman, 2008), and more recently the integration of information 
literacy training with problem solving in engineering assignments to enhance critical thinking (Lai & Loon, 2014). 

Challenges remain with incorporating critical thinking into engineering curricula and assessing the impact 
(Douglas, 2012) and (Bumbaco & Douglas, 2014), and it is not confined to the United States (Ahern, O’Connor, 
McRuairc, McNamara & O’Donnell, 2012). Ahern and colleagues challenged European engineering educators to 
engage more closely with educational literature to understand critical thinking and to explicitly address it in 
coursework. Claris and Riley (2012) challenged the current state of critical thinking teaching in engineering and 
developed “a reflexive view of critical thinking, leading to critical thinking not only within but also about 
engineering.” They found “that thinking critically about engineering can challenge power/knowledge relationships, 
critique engineering epistemologies, engage in reflective and reflexive practice, and work relationally for social 
justice.” Despite the challenges, educators agree that higher education needs to foster critical thinking deliberately 
and embrace it as a core value (Flores, Matkin, Burbach, Quinn, & Harding, 2012). 

METHODS 

The Engineering School embraced the Paul-Elder Framework as a means of introducing common labels 
and descriptions for all discussions of critical thinking and developed a program to inculcate critical thinking, 
overtly and intentionally, that extends from outreach programs to elementary and middle schools, throughout the 
undergraduate curriculum, to the culminating experience in engineering capstone design courses. In fall 2008, 
faculty at the Engineering School began a four-year educational research study to incorporate the Paul-Elder critical 
thinking framework intentionally and transparently, across the undergraduate engineering curriculum; they also 
developed a plan for assessing results of the program. The program operated as an ongoing feedback loop 
(assessment, revision, implementation, assessment, etc.) for evaluation of critical thinking skills, much as the 
ongoing Accreditation Board for Engineering and Technology (ABET) assessments function for engineering 
program curricula. 

The purpose of this study was to assess the development of critical thinking skills in undergraduate 
engineering students. The research question was, “How do the critical thinking skills of undergraduate engineering 
students change as they progress through the engineering program with the explicit and strategic incorporation of 
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critical thinking assignments?” The research hypothesis was, “There will be a significant increase in undergraduate 
students’ critical thinking abilities from the freshman to the senior year with the explicit and strategic incorporation 
of critical thinking assignments.” The study was a descriptive, longitudinal study of three engineering student 
cohorts as they progressed through the four year undergraduate program. The study was approved by the 
university’s Institutional Review Board. 


PARTICIPANTS/SUBJECTS 

Undergraduate engineering students were eligible to participate in the study if they were at least 18 years of 
age as freshmen and would complete the undergraduate engineering program in four years. A convenience sample of 
649 undergraduate engineering freshmen students, across the three study years, consented to participate in the study 
(57% consent rate). From the original consented sample, 182 students (28% of those who consented) completed the 
undergraduate engineering program in four years and are the unit of analysis for the longitudinal study. The greatest 
decrease in the study sample occurred from the freshman to sophomore year. This initial decrease was primarily due 
to first year attrition since students often transfer to other majors their first year. From the sophomore to the senior 
year, 75% of the consented students completed the undergraduate engineering program in four years. The continued 
decrease in sample size was primarily due to students changing majors within engineering failing to stay in the 
regular sequence of program courses due to dropping courses, not taking a full load, or other reasons. The 
demographics for each cohort are reported in Table 1. There was no statistically significant difference in high school 
GPA or ACT composite score across the 3 cohorts. 


Table 1. Demographics for the Three Study Cohorts 



Cohort 1 

Cohort 2 

Cohort 3 


(n=50) 

(n=62) 

(n=70) 

High School GPA Mean (SD) 

3.85 

3.83 

3.89 


(.170) 

(.222) 

(.220) 

ACT Composite Mean (SD) 

29.36 

29.36 

29.82 


(2.317) 

(3.188) 

(2.270) 

Gender 

Male 

40 

53 

55 

Female 

10 

9 

15 

Ethnicity 

White 

45 

49 

61 

Black 

0 

2 

0 

Hispanic 

0 

1 

2 

Asian 

2 

9 

1 

Other 

3 

1 

6 


DATA COLLECTION 


The study started with a core group of committed faculty who recognized that although most all 
engineering courses required students to think critically, faculty were not specifically teaching students how to do 
so. Faculty typically “demonstrated” critical thinking as they solved problems or discussed ethical or technical 
issues, rather than explicitly using a common language and framework to thoroughly discuss the “analyze and 
evaluate” process of critical thinking so clearly articulated in (Nosich, 2012). Engineering faculty developed or 
revised a critical thinking assignment for a selected course in each year of the undergraduate program. The 
assignments were authentic assessments embedded within the courses as part of the grading requirements and direct 
assessments of the students’ critical thinking abilities for course specific content. Student responses to the 
assignments (artifacts) were copied before grading, all identifying information was removed, and a unique study 
number was placed on the artifacts for tracking across the four-year study time period. Several faculty members in 
every department embraced the project to the extent that we are convinced every student was thoroughly exposed to 
the Paul-Elder critical thinking framework in at least one course per year. 
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The specific critical thinking assignment for a required course each year that all engineering students took 

was: 

Freshman: In the “Introduction to Engineering” course, a case-study was used to introduce various aspects of 
engineering before the Paul-Elder critical thinking framework was intentionally discussed in the course. A written 
summary of the case study served as the baseline or “pre” assessment. Details of the longitudinal study development 
analysis of the freshman course artifacts (baseline and course critical thinking assignments) and associated faculty 
scoring sessions for all three cohorts can be found in (Ralston, Larson, & Bays,2011). 

Sophomore: The “Differential Equations” course’s artifact was a one page assignment that required students to 
answer the question “Of the many methods for solving differential equations covered in this course, if you had time 
to teach only one method to someone, which method would you pick and why?” Faculty reviewed the Paul-Elder 
critical thinking framework in class before this assignment was submitted. 

Junior: The artifact collected from third year students was their cooperative internship report summary which asked 
students to critically reflect on their primary responsibilities. The cooperative internship seminar classes spent time 
formally reviewing the Paul-Elder critical thinking framework, as well as modeling for students how critical 
thinking skills could be used in interviewing and in engineering practice. 

Senior: Artifacts from required fourth year courses from the seven degree-granting departments were written 
summaries from either capstone design projects or other individual assignments requiring critical analysis. In the 
fourth year, some faculty teaching courses where artifacts were collected made overt efforts to teach the Paul-Elder 
critical thinking framework, and others simply provided review materials. 

HOLISTIC CRITICAL THINKING RUBRIC 

Although critical thinking and assessment of critical thinking using rubrics have been discussed in the 
engineering education literature (Berge &Flora, 2010; Goiter, Van Wie, Brown, Thiessen, Yurt, & Abdul, 2009; 
Ralston & Bays, 2010), the studies have primarily related to a single course, not complete curriculum redesign and 
assessment. No existing critical thinking rubric was found that incorporated the Paul-Elder critical thinking 
framework for engineering. The engineering faculty decided to construct a four-point, holistic critical thinking 
rubric based on the Paul-Elder critical thinking framework for use with the research project (Ralston & Bays, 2010). 
Figure 2 presents the engineering holistic critical thinking rubric. 
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_ Figure 2. Holistic Critical Thinking Rubric _ 

University of Louisville 

J.B. Speed School of Engineering 

Holistic Critical Thinking Rubric* 

Consistently does all or most of the following: 

4 Clearly identifies the purpose including all complexities of relevant questions. 

Accurate, complete information that is supported by relevant evidence. 

Complete, fair presentation of all relevant assumptions and points of view. 

Clearly articulates significant, logical implications and consequences based on relevant evidence. 

3 Clearly identifies the purpose including some complexities of relevant questions. 

Accurate, mostly complete information that is supported by evidence. 

Complete, fair presentation of some relevant assumptions and points of view. 

Clearly articulates some implications and consequences based on evidence. 

2 Identifies the purpose including irrelevant and/or insufficient questions. 

Accurate but incomplete information that is not supported by evidence. 

Simplistic presentation that ignores relevant assumptions and points of view. 

Articulates insignificant or illogical implications and consequences that are not supported by evidence. 

1 Unclear purpose that does not include questions. 

Inaccurate, incomplete information that is not supported by evidence. 

Incomplete presentation that ignores relevant assumptions and points of view. 

Fails to recognize or generates invalid implications and consequences based on irrelevant evidence. 

*Based on the Paul-Elder critical thinking framework _ 


A total of 20 engineering faculty volunteered to score the student artifacts throughout the research project. 
Faculty received a stipend to spend on any academic expense as a compensation for the time the scoring session 
would take over the multi-year length of the research project. Engineering faculty were trained to score the blinded 
artifacts using the holistic critical thinking rubric. At least two faculty independently scored each artifact. Each 
faculty rater was given a file with the blinded student artifacts, assignment description, critical thinking rubric, and a 
form to record their scores. If there was greater than a one point discrepancy between the two faculty rater scores, a 
third faculty scored the artifact. Final critical thinking scores for each artifact were determined by averaging the two 
or three blinded, independent faculty rater scores. The procedure for scoring the engineering student artifacts was 
similar to the procedure for scoring university-wide outcomes. The intraclass correlation coefficient (ICC) was used 
to assess the consistency of the paired faculty rater scores across the study assessment periods. The average ICC 
increased from 0.377 in the first session to 0.428 in the final session. The number of artifacts that necessitated a third 
rater decreased over time from 20% in the first session to 5% in the final session. Different faculty pairs rated the 
artifacts in the first and final sessions. 


FINDINGS 

Table 2 presents the means and standard deviations for the artifact analyzed from each cohort. Figure 3 
graphically illustrates the mean scores for the artifact analyzed from each cohort. 


Table 2. Cohort Critical Thinking Artifact Means and Standard Deviations 



Cohort 1 
(n=50) 

Cohort 2 
(n=62) 

Cohort 3 
(n=70) 

Freshman 

2.6434 (.71045) 

2.3871 (.63142) 

2.4000 (.59952) 

Sophomore 

2.6563 (.58860) 

2.3846 (.59945) 

2.3620 (.59048) 

Junior 

2.8370 (.57898) 

2.9781 (.59432) 

2.8833 (.68061) 

Senior 

3.1367 (.63504) 

2.9437 (.65337) 

2.9596 (.67292) 
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Figure 3. Cohort Critical Thinking Artifact Mean Scores_ 
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The freshman, sophomore and senior means for cohort 1 were higher than those for cohort 2 and 3. The 
freshman, sophomore, junior and senior means for cohort 2 and cohort 3 were similar in value and trajectory. The 
sophomore artifact average score was significantly (p<,05) higher for cohort 1 when compared to cohort 2 and 
cohort 3. There was no statistically significant difference (p>.05) between the three cohort groups for any of the 
remaining artifact average scores. 


A repeated measures ANOVA was used to assess the change in subject’s critical thinking scores over time. 
The assumption of sphericity, relatively equal variances, was evidenced by Mauchly’s Test of Sphericity so no 
corrections were applied to the F-ratio computations. Table 3 presents the sphericity calculations for each cohort. 


Table 3. Cohort Sphericity for Repeated Measures ANOVA 



Cohort 1 
(n=50) 

Cohort 2 
(n=62) 

Cohort 3 
(n=70) 

Mauchly’s W 

.893 

.935 

.940 

Chi-Square (df) 

4.715(5) 

3.9969 (5) 

4.155(5) 

Significance 

.452 

.554 

.527 


There was a statistically significant increase in critical thinking scores over the four years. Table 4 contains 
the repeated measures ANOVA for the three cohorts. 


Table 4. Repeated Measures ANOVA for the Three Cohorts 



Cohort 1 
(n=50) 

Cohort 2 
(n=62) 

Cohort 3 
(n=70) 

Sum of Squares 

8.616 

20.247 

20.699 

Mean Squares 

2.872 

6.749 

6.90 

F (df) 

6.693 (3) 

20.254 (3) 

20.611 (3) 

Significance 

.000 

.000 

.000 


Specifically, the critical thinking score for the senior artifact was significantly higher than the freshman (p 
<.05) and the sophomore (p <,05) critical thinking scores for all three cohorts. The critical thinking score for the 
junior artifact was significantly higher than the freshman (p <,05) and the sophomore (p <.05) critical thinking 
scores for cohort 2 and cohort 3. Tables 5, 6, and 7 contain the Bonferroni comparisons used to identify these 
statistically significant different critical thinking artifact scores in each of the three cohorts. 
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Table 5. Bonferroni Comparison of Critical Thinking Scores in Cohort 1 






95% Cl 

Comparisons 

Mean Score 
Difference 

Std. Error 

Sig. 

Lower Bound 

Upper Bound 

Senior vs. Freshman 

.553 

.152 

.004 

.133 

.973 

Senior vs. Sophomore 

.527 

.135 

.002 

.152 

.901 

Senior vs. Junior 

.318 

.140 

.169 

-.069 

.706 

Junior vs. Freshman 

.235 

.147 

.707 

-.172 

.641 

Junior vs. Sophomore 

.208 

.113 

.435 

-.105 

.521 

Sophomore vs. Freshman 

.026 

.147 

1.00 

-.380 

.433 


Table 6. Bonferroni Comparison of Critical Thinking Scores in Cohort 2 


Comparisons 

Mean Score 
Difference 

Std. Error 

Sig. 

95% Cl 

Lower Bound 

Upper Bound 

Senior vs. Freshman 

.558 

.113 

.000 

.249 

.867 

Senior vs. Sophomore 

.588 

.102 

.000 

.280 

.837 

Senior vs. Junior 

-.035 

.112 

1.00 

-.340 

.269 

Junior vs. Freshman 

.593 

.108 

.000 

.298 

.887 

Junior vs. Sophomore 

.594 

.098 

.000 

.326 

.861 

Sophomore vs. Freshman 

-.001 

.092 

1.00 

-.253 

.252 


Table 7. Bonferroni Comparison of Critical Thinking Scores in Cohort 3 






95% Cl 

Comparisons 

Mean Score 
Difference 

Std. Error 

Sig. 

Lower Bound 

Upper Bound 

Senior vs. Freshman 

.560 

.091 

.000 

.312 

.807 

Senior vs. Sophomore 

.598 

.099 

.000 

.329 

.866 

Senior vs. Junior 

.076 

.110 

1.00 

-.222 

.374 

Junior vs. Freshman 

.483 

.094 

.000 

.227 

.739 

Junior vs. Sophomore 

.521 

TOO 

.000 

.249 

.794 

Sophomore vs. Freshman 

-.038 

.091 

1.00 

-.286 

.210 


DISCUSSION 


The significant increase in critical thinking scores over the four years supported the research hypothesis. 
Although there is not a baseline cohort for comparison who did not receive explicit critical thinking instruction, the 
increase in critical thinking scores is an encouraging finding for the engineering profession and public who need 
engineers who can think critically. Some may challenge tracking critical thinking assessment using specific course 
content due to the fact that the artifact rated is different in each course, as opposed to implementing a uniform 
critical thinking test, although one study attests to the superiority of assessing specific course content (Renaud & 
Murray, 2008). The approach can be further justified by the literature since it supplied exposure and discussion in 
many contexts, each artifact was a written analysis of some type of problem-solving, and the assignments placed a 
continued emphasis on critical thinking as a needed meta-cognitive skill, all of which have been cited as important 
in critical thinking skill development (Bailin, Case, Coombs & Daniels, 1999), (Tsui, 2000, 2002), Cooney et al. 
(2008). The participating faculty shared with the study authors that their intentional focus and emphasis on teaching 
critical thinking improved their teaching as they specifically emphasized meta- cognition to students. The faculty 
felt they encouraged purposeful, reasoned, and goal-directed thinking as discussed by Halpern (2003); that 
intentionally working with students to become meta-cognitive has value. 

This study tracked only those students who stayed in-phase with their freshman year cohort. For example, 
if a student got out of sequence with some of the pre-requisites, they might take the sophomore course a different 
semester from the main semester tracked; or a student might go on cooperative internship in a semester off from 
their cohort. For this reason, as well as attrition from the engineering school in general, the numbers dropped from 
year one to year four for each cohort. However, as infusing critical thinking was new to the engineering school 
during this project and resources were scarce, key faculty teaching in-phase cohort courses were the only ones 
targeted for this study. 
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The decrease from cohort 1 to cohorts 2 and 3 in the faculty rater average CT score is most likely the result 
of faculty gaining experience in scoring against a holistic rubric. It took faculty some time to learn to avoid norm 
referencing and to avoid trying to compensate for the age of the student. As faculty gained practice and experience 
scoring with the rubric, they became much more aware of the need for students to support their statements rather 
than accepting reasonable statements without proper justification. As faculty began to internalize the components of 
the framework, especially the use of the standards to assess the quality of the reasoning, they tended to give more 
low scores and fewer high scores. We also noticed that fatigue played a huge role in scoring consistency as did 
content-knowledge level of the rater. Faculty noted that distinguishing between 2’s and 3’s were often difficult. 
They expressed some preference for modifying the rubric for future projects. 

Limitations of the study include the convenience sample, lack of a control group, study participant attrition, 
and use of an institution developed assessment rubric. The authors acknowledge the absence of a control group 
compromises the study design. However, the engineering program had already begun to implement critical thinking 
strategies as part of the university’s quality enhancement plan for regional reaccreditation. To include a control 
group would have delayed the program’s progression on implementation of the university-wide initiative. 

CONCLUSION/SUMMARY 

The longitudinal study provided an opportunity to encourage and support faculty in the intentional and 
transparent teaching of critical thinking, and demonstrated a positive impact on the development of students’ critical 
thinking abilities. Faculty embraced the Paul-Elder framework and produced a holistic rubric for engineering based 
on the framework. Faculty raters continue to refine and enhance course critical thinking experiences. There are 
dedicated and sustained efforts in the school of engineering to embed in an intentional and systematic manner 
critical thinking instruction and assignments in core courses that align with the Paul-Elder critical thinking 
framework. This type of assessment strengthens and augments ongoing strategies that assess ABET outcomes, and 
this instruction and assessment should continue to improve the critical thinking skills of undergraduates which is 
considered highly important by college faculty, administrators and employers alike (Stein & Haynes, 2011), 
(AAC&U,2011), (Hart Research Study, 2013). 

This model for developing assignments and assessing student responses fills a gap that is in the literature 
for engineering program assessment of critical thinking and can be duplicated in other disciplines interested in 
program assessment of critical thinking. 

As a result of this longitudinal study, faculty in the engineering school have anecdotally reported placing 
more time and intentional emphasis on the teaching of critical thinking. Faculty had always demonstrated critical 
thinking, but with the introduction of the Paul-Elder critical thinking framework and collection of written artifacts 
that require an explanation of the thinking involved in problem solving, the undergraduates are getting a more 
focused, explicit, intentional emphasis on critical thinking that will be invaluable in their engineering practice. 
Data are being analyzed from a follow-up study that assessed the impact of this study on faculty participants, 
specifically the impact of the rating sessions and training on their teaching of critical thinking. 

ADDITIONAL PAPER INFORMATION 

This paper presents the final analysis and results of a six year longitudinal study that followed three cohorts 
of students through their undergraduate program. Initial discussion and preliminary results have been presented in 
previous papers by the authors (Ralston & Bays, 2010; Ralston, Larson, & Bays, 2011; Ralston & Bays, 2013). 
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